A variety of sophisticated systems have been developed for monitoring and forecasting performance in various fields. These monitoring and forecasting systems may be referred to, collectively or individually, as “estimation systems.” For example, conventional statistics systems, artificial neural network systems, or concurrent learning and information processing (CLIP) systems capable of monitoring and forecasting a large number of variables have been developed for use in a variety of fields including computer performance monitoring and forecasting, visual image processing, electricity demand forecasting, and commodity price forecasting. These estimation systems typically use a number of measured input values to impute (i.e., estimate for a current time trial) current values for monitoring, and they may also predict (i.e., estimate for future time trials) future input values. In particular, these systems may compare imputed input values with actual input values to identify abnormal input values when they occur, and they may also predict or forecast the likelihood that future input values will become abnormal.
The mathematical core technology of these monitoring and forecasting systems may involve the computation of a matrix of estimation parameters, also called learned parameters. This matrix typically contains observed relationships, such as the covariances, between input and output variables. Estimation systems may also utilize the inverse of the covariance matrix, which is sometimes referred to as a “connection weight matrix.” In particular, the elements of the covariance matrix are typically estimated through the application of statistical analysis to a historical database of input and output values. Once the covariance matrix has been computed, it may be inverted to obtain the connection weight matrix, which can be used to directly compute estimates for the output values from a given set of input values through standard matrix computations.
Moreover, once the covariance and connection weight matrices have been initially determined, they may be used to perform monitoring and forecasting on a CLIP basis. That is, a new set of computed output values may be computed for each time trial of measured input values, and for each time trial either the covariance matrix, the connection weight matrix, or both may be updated to achieve learning while the system performs its monitoring and prediction functions. See in particular U.S. Pat. No. 5,835,902.
With these systems, certain estimation problems may exist initially, while others may develop over time. For example, some input variables may not be predictive of output variables initially, while others that are predictive initially may not be predictive later. In addition, some input variables may be linearly redundant initially while others may become linearly redundant later.
Some of these problems may be solved in order to improve estimation efficiency while others should be solved in order to prevent numerical errors. For example, if many useless inputs are included in an estimation system, estimation accuracy degrades significantly. As a second example, if linearly redundant inputs exist initially or develop over time, computer processors may attempt to divide by zero, which may cause computer operations to break down. To avoid these problems, historical data may be continually reevaluated to ensure that the most effective input values and estimation algorithms are in use. This process is known as model refinement.
Traditionally, model refinement operations have been conducted by expert analysts, using representative data in a historical sample. Such operations typically take considerable effort and time, which may be acceptable in settings where representative datasets are easy to obtain and estimation conditions do not change rapidly. However, such operations may be impractical or impossible in settings where information is arriving quickly and conditions are changing rapidly. One such setting is application performance monitoring (APM) services, where computer monitoring variables, such as computer usage and input-output rates, fluctuate systematically on a daily basis or even on a sub-hourly basis. A continuing need therefore exists for effective and efficient model refinement systems, and in particular for systems that simplify and automate the model refinement process.
In addition to the model refinement need, a second need exists to refine the sensitivity of monitoring systems. The information produced by an estimation system can be used to identify detected or predicted problems only when the information produced by the estimation system is effectively communicated to system administrators. Typically, a system of alerts and alarms may be used to notify the system administrators when the estimation system detects a problem. However, the sensitivity of the monitored system to the measured input values can change over time. Unless the alert and alarm levels are adjusted to reflect these underlying system changes, they can become inappropriate, for example by producing too many false alarms or by producing too few valid alarms. To avoid these problems, reporting indicators such as alert and alarm levels may be continually reevaluated to ensure that the most effective alert and alarm thresholds are in use. This process is known as reporting refinement. A continuing need exists for effective and efficient reporting refinement systems, and in particular for systems-that simplify and automate the reporting refinement process.
Therefore, a continuing need exists for more effective, useful, automated, and understandable systems for configuring and monitoring estimation systems generally, and computer performance estimation systems in particular. In addition, a continuing need exists for more effective model refinement and reporting refinement systems.